2024-12-16 09:41:56,934 [ 78846 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2024-12-16 09:41:56,935 [ 78846 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2024-12-16 09:41:56,935 [ 78846 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2024-12-16 09:41:56,935 [ 78846 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_c5xcdm --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1d1c87eee2db -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_allowed_client_hosts/test.py::test_allowed_host test_allowed_client_hosts/test.py::test_denied_host test_async_load_databases/test.py::test_dependent_tables test_async_load_databases/test.py::test_dict_get_data test_async_load_databases/test.py::test_multiple_tables 'test_attach_backup_from_s3_plain/test.py::test_attach_part[compact0]' 'test_attach_backup_from_s3_plain/test.py::test_attach_part[compact1]' 'test_attach_backup_from_s3_plain/test.py::test_attach_part[wide0]' 'test_attach_backup_from_s3_plain/test.py::test_attach_part[wide1]' test_attach_table_from_s3_plain_readonly/test.py::test_attach_table_from_s3_plain_readonly test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart test_authentication/test.py::test_authentication_fail test_authentication/test.py::test_authentication_pass test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True]' test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes test_backup_restore_on_cluster/test.py::test_empty_replicated_table test_backup_restore_on_cluster/test.py::test_file_deduplication test_backup_restore_on_cluster/test.py::test_get_error_from_other_host test_backup_restore_on_cluster/test.py::test_keeper_value_max_size test_backup_restore_on_cluster/test.py::test_mutation test_backup_restore_on_cluster/test.py::test_projection test_backup_restore_on_cluster/test.py::test_replicated_database test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts test_backup_restore_on_cluster/test.py::test_replicated_table test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath test_backup_restore_on_cluster/test.py::test_required_privileges 'test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False]' 'test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True]' test_backup_restore_on_cluster/test.py::test_system_functions test_backup_restore_on_cluster/test.py::test_system_users test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty test_backup_restore_on_cluster/test.py::test_tables_dependency test_backward_compatibility/test_cte_distributed.py::test_cte_distributed test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts test_block_structure_mismatch/test.py::test test_broken_projections/test.py::test_broken_ignored test_broken_projections/test.py::test_broken_ignored_replicated test_broken_projections/test.py::test_broken_projections_in_backups_1 test_broken_projections/test.py::test_broken_projections_in_backups_2 test_broken_projections/test.py::test_broken_projections_in_backups_3 test_broken_projections/test.py::test_materialize_broken_projection test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile test_cgroup_limit/test.py::test_cgroup_cpu_limit 'test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node]' 'test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries]' test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd]' test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True]' test_cleanup_after_start/test.py::test_old_dirs_cleanup test_concurrent_queries_restriction_by_query_kind/test.py::test_insert test_concurrent_queries_restriction_by_query_kind/test.py::test_select test_config_hide_in_preprocessed/test.py::test_hide_in_preprocessed test_config_xml_main/test.py::test_xml_main_conf test_covered_by_broken_exists/test.py::test_make_clone_covered_by_broken_detached_dir_exists test_default_database_on_cluster/test.py::test_default_database_on_cluster test_default_role/test.py::test_alter_user test_default_role/test.py::test_set_default_roles test_default_role/test.py::test_wrong_set_default_role test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed]' -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: order-1.0.1, xdist-3.5.0, reportlog-0.4.0, random-0.2, repeat-0.9.3, timeout-2.2.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] test_broken_projections/test.py::test_broken_ignored test_default_role/test.py::test_alter_user test_attach_backup_from_s3_plain/test.py::test_attach_part[compact0] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] test_attach_table_normalizer/test.py::test_attach_substr test_async_load_databases/test.py::test_dependent_tables test_authentication/test.py::test_authentication_fail test_allowed_client_hosts/test.py::test_allowed_host [gw9] [ 1%] PASSED test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart [gw8] [ 2%] PASSED test_authentication/test.py::test_authentication_fail test_authentication/test.py::test_authentication_pass [gw8] [ 3%] PASSED test_authentication/test.py::test_authentication_pass [gw6] [ 4%] PASSED test_default_role/test.py::test_alter_user test_default_role/test.py::test_set_default_roles [gw6] [ 5%] PASSED test_default_role/test.py::test_set_default_roles test_default_role/test.py::test_wrong_set_default_role [gw6] [ 6%] PASSED test_default_role/test.py::test_wrong_set_default_role [gw9] [ 7%] PASSED test_attach_table_normalizer/test.py::test_attach_substr_restart [gw2] [ 8%] PASSED test_attach_backup_from_s3_plain/test.py::test_attach_part[compact0] test_attach_backup_from_s3_plain/test.py::test_attach_part[compact1] [gw2] [ 9%] PASSED test_attach_backup_from_s3_plain/test.py::test_attach_part[compact1] test_attach_backup_from_s3_plain/test.py::test_attach_part[wide0] [gw2] [ 10%] PASSED test_attach_backup_from_s3_plain/test.py::test_attach_part[wide0] test_attach_backup_from_s3_plain/test.py::test_attach_part[wide1] [gw2] [ 11%] PASSED test_attach_backup_from_s3_plain/test.py::test_attach_part[wide1] test_concurrent_queries_restriction_by_query_kind/test.py::test_insert test_attach_table_from_s3_plain_readonly/test.py::test_attach_table_from_s3_plain_readonly test_buffer_profile/test.py::test_buffer_profile [gw7] [ 12%] PASSED test_allowed_client_hosts/test.py::test_allowed_host test_allowed_client_hosts/test.py::test_denied_host [gw7] [ 13%] PASSED test_allowed_client_hosts/test.py::test_denied_host [gw5] [ 14%] PASSED test_async_load_databases/test.py::test_dependent_tables test_async_load_databases/test.py::test_dict_get_data test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible [gw4] [ 15%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] [gw3] [ 16%] PASSED test_broken_projections/test.py::test_broken_ignored test_broken_projections/test.py::test_broken_ignored_replicated [gw0] [ 17%] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] [gw3] [ 18%] PASSED test_broken_projections/test.py::test_broken_ignored_replicated test_broken_projections/test.py::test_broken_projections_in_backups_1 test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] [gw0] [ 19%] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] [gw9] [ 20%] PASSED test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile [gw9] [ 21%] PASSED test_buffer_profile/test.py::test_default_profile [gw3] [ 22%] PASSED test_broken_projections/test.py::test_broken_projections_in_backups_1 test_broken_projections/test.py::test_broken_projections_in_backups_2 [gw0] [ 23%] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] [gw8] [ 24%] PASSED test_concurrent_queries_restriction_by_query_kind/test.py::test_insert test_concurrent_queries_restriction_by_query_kind/test.py::test_select [gw1] [ 25%] PASSED test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica [gw0] [ 26%] PASSED test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes [gw3] [ 27%] PASSED test_broken_projections/test.py::test_broken_projections_in_backups_2 test_broken_projections/test.py::test_broken_projections_in_backups_3 [gw0] [ 28%] PASSED test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes [gw4] [ 29%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] test_cgroup_limit/test.py::test_cgroup_cpu_limit test_backup_restore_on_cluster/test.py::test_empty_replicated_table [gw0] [ 30%] PASSED test_backup_restore_on_cluster/test.py::test_empty_replicated_table test_backup_restore_on_cluster/test.py::test_file_deduplication [gw3] [ 31%] PASSED test_broken_projections/test.py::test_broken_projections_in_backups_3 test_broken_projections/test.py::test_materialize_broken_projection [gw0] [ 32%] PASSED test_backup_restore_on_cluster/test.py::test_file_deduplication test_backward_compatibility/test_cte_distributed.py::test_cte_distributed test_backup_restore_on_cluster/test.py::test_get_error_from_other_host [gw0] [ 33%] PASSED test_backup_restore_on_cluster/test.py::test_get_error_from_other_host [gw3] [ 34%] PASSED test_broken_projections/test.py::test_materialize_broken_projection test_backup_restore_on_cluster/test.py::test_keeper_value_max_size [gw1] [ 35%] PASSED test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries [gw0] [ 36%] PASSED test_backup_restore_on_cluster/test.py::test_keeper_value_max_size [gw1] [ 37%] PASSED test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works [gw6] [ 38%] PASSED test_attach_table_from_s3_plain_readonly/test.py::test_attach_table_from_s3_plain_readonly test_backup_restore_on_cluster/test.py::test_mutation test_default_database_on_cluster/test.py::test_default_database_on_cluster [gw1] [ 39%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] [gw1] [ 40%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] [gw1] [ 41%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] [gw1] [ 42%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] [gw4] [ 43%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] [gw0] [ 44%] PASSED test_backup_restore_on_cluster/test.py::test_mutation [gw5] [ 45%] PASSED test_async_load_databases/test.py::test_dict_get_data test_async_load_databases/test.py::test_multiple_tables [gw1] [ 46%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] test_backup_restore_on_cluster/test.py::test_projection [gw1] [ 47%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] [gw7] [ 48%] PASSED test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible [gw1] [ 49%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] [gw7] [ 50%] PASSED test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible [gw2] [ 51%] PASSED test_backward_compatibility/test_cte_distributed.py::test_cte_distributed [gw0] [ 52%] PASSED test_backup_restore_on_cluster/test.py::test_projection [gw1] [ 53%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] test_backup_restore_on_cluster/test.py::test_replicated_database [gw1] [ 54%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] [gw9] [ 55%] PASSED test_cgroup_limit/test.py::test_cgroup_cpu_limit test_config_xml_main/test.py::test_xml_main_conf [gw8] [ 56%] PASSED test_concurrent_queries_restriction_by_query_kind/test.py::test_select [gw4] [ 57%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] [gw0] [ 58%] FAILED test_backup_restore_on_cluster/test.py::test_replicated_database [gw1] [ 59%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column test_config_hide_in_preprocessed/test.py::test_hide_in_preprocessed [gw1] [ 60%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] [gw5] [ 61%] PASSED test_async_load_databases/test.py::test_multiple_tables test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts test_detached_parts_metrics/test.py::test_numbers_of_detached_parts [gw1] [ 62%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] [gw0] [ 63%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts test_cleanup_after_start/test.py::test_old_dirs_cleanup [gw1] [ 64%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] [gw7] [ 65%] PASSED test_config_hide_in_preprocessed/test.py::test_hide_in_preprocessed [gw0] [ 66%] FAILED test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts [gw2] [ 67%] PASSED test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column test_backup_restore_on_cluster/test.py::test_replicated_table [gw0] [ 68%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster [gw1] [ 69%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] [gw0] [ 70%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster [gw1] [ 71%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster [gw9] [ 72%] PASSED test_config_xml_main/test.py::test_xml_main_conf [gw0] [ 73%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup [gw3] [ 74%] PASSED test_default_database_on_cluster/test.py::test_default_database_on_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def [gw1] [ 75%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] [gw0] [ 76%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert [gw1] [ 77%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried [gw1] [ 78%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried [gw0] [ 79%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge [gw0] [ 80%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath [gw4] [ 81%] PASSED test_detached_parts_metrics/test.py::test_numbers_of_detached_parts [gw0] [ 82%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath test_backup_restore_on_cluster/test.py::test_required_privileges [gw1] [ 83%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] [gw0] [ 84%] PASSED test_backup_restore_on_cluster/test.py::test_required_privileges test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] [gw5] [ 85%] PASSED test_cleanup_after_start/test.py::test_old_dirs_cleanup [gw1] [ 86%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] [gw0] [ 87%] PASSED test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] [gw8] [ 88%] PASSED test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts [gw0] [ 89%] PASSED test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] test_backup_restore_on_cluster/test.py::test_system_functions [gw0] [ 90%] PASSED test_backup_restore_on_cluster/test.py::test_system_functions test_covered_by_broken_exists/test.py::test_make_clone_covered_by_broken_detached_dir_exists test_backup_restore_on_cluster/test.py::test_system_users [gw0] [ 91%] PASSED test_backup_restore_on_cluster/test.py::test_system_users test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def [gw0] [ 92%] PASSED test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty [gw0] [ 93%] PASSED test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty test_backup_restore_on_cluster/test.py::test_tables_dependency [gw1] [ 94%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] [gw0] [ 95%] PASSED test_backup_restore_on_cluster/test.py::test_tables_dependency [gw6] [ 96%] PASSED test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup [gw1] [ 97%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] test_block_structure_mismatch/test.py::test [gw8] [ 98%] PASSED test_covered_by_broken_exists/test.py::test_make_clone_covered_by_broken_detached_dir_exists [gw1] [ 99%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] [gw6] [100%] PASSED test_block_structure_mismatch/test.py::test =================================== FAILURES =================================== ___________________________ test_replicated_database ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 def test_replicated_database(): node1.query( "CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}')" ) node1.query( "CREATE TABLE mydb.tbl(x UInt8, y String) ENGINE=ReplicatedMergeTree ORDER BY x" ) node2.query("SYSTEM SYNC DATABASE REPLICA mydb") node1.query("INSERT INTO mydb.tbl VALUES (1, 'Don''t')") node2.query("INSERT INTO mydb.tbl VALUES (2, 'count')") node1.query("INSERT INTO mydb.tbl VALUES (3, 'your')") node2.query("INSERT INTO mydb.tbl VALUES (4, 'chickens')") node1.query("OPTIMIZE TABLE mydb.tbl ON CLUSTER 'cluster' FINAL") node1.query("SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl") # check data in sync expect = TSV([[1, "Don\\'t"], [2, "count"], [3, "your"], [4, "chickens"]]) assert node1.query("SELECT * FROM mydb.tbl ORDER BY x") == expect assert node2.query("SELECT * FROM mydb.tbl ORDER BY x") == expect # Make backup. backup_name = new_backup_name() node1.query( f"BACKUP DATABASE mydb ON CLUSTER 'cluster' TO {backup_name} SETTINGS replica_num=2" ) # Drop table on both nodes. node1.query("DROP DATABASE mydb ON CLUSTER 'cluster' SYNC") # Restore from backup on node2. node1.query(f"RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM {backup_name}") node1.query("SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl") > assert node1.query("SELECT * FROM mydb.tbl ORDER BY x") == expect E AssertionError: assert '' == 1 Don\'t\n2 count\n3 your\n4 chickens E + where '' = >('SELECT * FROM mydb.tbl ORDER BY x') E + where > = .query test_backup_restore_on_cluster/test.py:188: AssertionError ------------------------------ Captured log call ------------------------------- 2024-12-16 09:42:56 [ 558 ] DEBUG : Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}') on node1 (cluster.py:3450, query) 2024-12-16 09:42:56 [ 558 ] DEBUG : Executing query CREATE TABLE mydb.tbl(x UInt8, y String) ENGINE=ReplicatedMergeTree ORDER BY x on node1 (cluster.py:3450, query) 2024-12-16 09:42:56 [ 558 ] DEBUG : Executing query SYSTEM SYNC DATABASE REPLICA mydb on node2 (cluster.py:3450, query) 2024-12-16 09:42:56 [ 558 ] DEBUG : Executing query INSERT INTO mydb.tbl VALUES (1, 'Don''t') on node1 (cluster.py:3450, query) 2024-12-16 09:42:56 [ 558 ] DEBUG : Executing query INSERT INTO mydb.tbl VALUES (2, 'count') on node2 (cluster.py:3450, query) 2024-12-16 09:42:56 [ 558 ] DEBUG : Executing query INSERT INTO mydb.tbl VALUES (3, 'your') on node1 (cluster.py:3450, query) 2024-12-16 09:42:56 [ 558 ] DEBUG : Executing query INSERT INTO mydb.tbl VALUES (4, 'chickens') on node2 (cluster.py:3450, query) 2024-12-16 09:42:56 [ 558 ] DEBUG : Executing query OPTIMIZE TABLE mydb.tbl ON CLUSTER 'cluster' FINAL on node1 (cluster.py:3450, query) 2024-12-16 09:42:57 [ 558 ] DEBUG : Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl on node1 (cluster.py:3450, query) 2024-12-16 09:42:57 [ 558 ] DEBUG : Executing query SELECT * FROM mydb.tbl ORDER BY x on node1 (cluster.py:3450, query) 2024-12-16 09:42:57 [ 558 ] DEBUG : Executing query SELECT * FROM mydb.tbl ORDER BY x on node2 (cluster.py:3450, query) 2024-12-16 09:42:57 [ 558 ] DEBUG : Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster' TO Disk('backups', '12') SETTINGS replica_num=2 on node1 (cluster.py:3450, query) 2024-12-16 09:42:57 [ 558 ] DEBUG : Executing query DROP DATABASE mydb ON CLUSTER 'cluster' SYNC on node1 (cluster.py:3450, query) 2024-12-16 09:42:57 [ 558 ] DEBUG : Executing query RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM Disk('backups', '12') on node1 (cluster.py:3450, query) 2024-12-16 09:42:58 [ 558 ] DEBUG : Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl on node1 (cluster.py:3450, query) 2024-12-16 09:42:58 [ 558 ] DEBUG : Executing query SELECT * FROM mydb.tbl ORDER BY x on node1 (cluster.py:3450, query) ---------------------------- Captured log teardown ----------------------------- 2024-12-16 09:42:58 [ 558 ] DEBUG : Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 (cluster.py:3450, query) 2024-12-16 09:42:58 [ 558 ] DEBUG : Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 (cluster.py:3450, query) 2024-12-16 09:42:58 [ 558 ] DEBUG : Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 (cluster.py:3450, query) 2024-12-16 09:42:59 [ 558 ] DEBUG : Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 (cluster.py:3450, query) ____________________ test_replicated_database_compare_parts ____________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 def test_replicated_database_compare_parts(): """ stop merges and fetches then write data to two nodes and compare that parts are restored from single node (second) after backup replica is selected by settings replica_num=2, replica_num_in_backup=2 """ node1.query( "CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}')" ) node1.query( "CREATE TABLE mydb.tbl(x UInt8, y String) ENGINE=ReplicatedMergeTree ORDER BY x" ) node2.query("SYSTEM SYNC DATABASE REPLICA mydb") node1.query("SYSTEM STOP MERGES mydb.tbl") node2.query("SYSTEM STOP MERGES mydb.tbl") node1.query("SYSTEM STOP FETCHES mydb.tbl") node2.query("SYSTEM STOP FETCHES mydb.tbl") node1.query("INSERT INTO mydb.tbl VALUES (1, 'a')") node1.query("INSERT INTO mydb.tbl VALUES (2, 'b')") node2.query("INSERT INTO mydb.tbl VALUES (3, 'x')") node2.query("INSERT INTO mydb.tbl VALUES (4, 'y')") p2 = node2.query("SELECT * FROM mydb.tbl ORDER BY x") # Make backup. backup_name = new_backup_name() node1.query( f"BACKUP DATABASE mydb ON CLUSTER 'cluster' TO {backup_name} SETTINGS replica_num=2" ) # Drop table on both nodes. node1.query("DROP DATABASE mydb ON CLUSTER 'cluster' SYNC") # Restore from backup on node2. node1.query( f"RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM {backup_name} SETTINGS replica_num_in_backup=2" ) node1.query("SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl") # compare parts p1_ = node1.query("SELECT _part, * FROM mydb.tbl ORDER BY x") p2_ = node2.query("SELECT _part, * FROM mydb.tbl ORDER BY x") assert p1_ == p2_ # compare data > assert p2 == node2.query("SELECT * FROM mydb.tbl ORDER BY x") E AssertionError: assert '3\tx\n4\ty\n' == '4\ty\n' E + 3 x E 4 y test_backup_restore_on_cluster/test.py:243: AssertionError ------------------------------ Captured log call ------------------------------- 2024-12-16 09:43:03 [ 558 ] DEBUG : Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}') on node1 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query CREATE TABLE mydb.tbl(x UInt8, y String) ENGINE=ReplicatedMergeTree ORDER BY x on node1 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query SYSTEM SYNC DATABASE REPLICA mydb on node2 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query SYSTEM STOP MERGES mydb.tbl on node1 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query SYSTEM STOP MERGES mydb.tbl on node2 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query SYSTEM STOP FETCHES mydb.tbl on node1 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query SYSTEM STOP FETCHES mydb.tbl on node2 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query INSERT INTO mydb.tbl VALUES (1, 'a') on node1 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query INSERT INTO mydb.tbl VALUES (2, 'b') on node1 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query INSERT INTO mydb.tbl VALUES (3, 'x') on node2 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query INSERT INTO mydb.tbl VALUES (4, 'y') on node2 (cluster.py:3450, query) 2024-12-16 09:43:04 [ 558 ] DEBUG : Executing query SELECT * FROM mydb.tbl ORDER BY x on node2 (cluster.py:3450, query) 2024-12-16 09:43:05 [ 558 ] DEBUG : Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster' TO Disk('backups', '14') SETTINGS replica_num=2 on node1 (cluster.py:3450, query) 2024-12-16 09:43:05 [ 558 ] DEBUG : Executing query DROP DATABASE mydb ON CLUSTER 'cluster' SYNC on node1 (cluster.py:3450, query) 2024-12-16 09:43:05 [ 558 ] DEBUG : Executing query RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM Disk('backups', '14') SETTINGS replica_num_in_backup=2 on node1 (cluster.py:3450, query) 2024-12-16 09:43:05 [ 558 ] DEBUG : Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl on node1 (cluster.py:3450, query) 2024-12-16 09:43:05 [ 558 ] DEBUG : Executing query SELECT _part, * FROM mydb.tbl ORDER BY x on node1 (cluster.py:3450, query) 2024-12-16 09:43:05 [ 558 ] DEBUG : Executing query SELECT _part, * FROM mydb.tbl ORDER BY x on node2 (cluster.py:3450, query) 2024-12-16 09:43:05 [ 558 ] DEBUG : Executing query SELECT * FROM mydb.tbl ORDER BY x on node2 (cluster.py:3450, query) ---------------------------- Captured log teardown ----------------------------- 2024-12-16 09:43:06 [ 558 ] DEBUG : Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 (cluster.py:3450, query) 2024-12-16 09:43:06 [ 558 ] DEBUG : Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 (cluster.py:3450, query) 2024-12-16 09:43:06 [ 558 ] DEBUG : Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 (cluster.py:3450, query) 2024-12-16 09:43:06 [ 558 ] DEBUG : Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 (cluster.py:3450, query) ============================== slowest durations =============================== 34.39s call test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 30.55s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 29.73s setup test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] 26.04s setup test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 25.18s setup test_attach_table_from_s3_plain_readonly/test.py::test_attach_table_from_s3_plain_readonly 24.69s call test_async_load_databases/test.py::test_dict_get_data 23.61s teardown test_attach_table_from_s3_plain_readonly/test.py::test_attach_table_from_s3_plain_readonly 23.39s teardown test_attach_backup_from_s3_plain/test.py::test_attach_part[wide1] 23.07s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] 21.90s setup test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 21.87s setup test_default_database_on_cluster/test.py::test_default_database_on_cluster 19.93s setup test_allowed_client_hosts/test.py::test_allowed_host 19.62s call test_concurrent_queries_restriction_by_query_kind/test.py::test_select 19.33s setup test_broken_projections/test.py::test_broken_ignored 18.60s setup test_covered_by_broken_exists/test.py::test_make_clone_covered_by_broken_detached_dir_exists 18.24s setup test_cleanup_after_start/test.py::test_old_dirs_cleanup 17.96s setup test_attach_backup_from_s3_plain/test.py::test_attach_part[compact0] 16.42s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] 15.76s call test_cgroup_limit/test.py::test_cgroup_cpu_limit 15.46s call test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 14.59s call test_config_xml_main/test.py::test_xml_main_conf 14.54s setup test_block_structure_mismatch/test.py::test 14.45s setup test_async_load_databases/test.py::test_dependent_tables 14.41s setup test_authentication/test.py::test_authentication_fail 13.91s setup test_default_role/test.py::test_alter_user 13.88s setup test_attach_table_normalizer/test.py::test_attach_substr 13.75s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] 13.52s setup test_buffer_profile/test.py::test_buffer_profile 12.83s setup test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 12.72s setup test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 12.41s call test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] 12.23s call test_async_load_databases/test.py::test_dependent_tables 12.07s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] 11.95s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] 11.89s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] 11.82s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] 11.63s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] 10.93s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] 9.89s setup test_backward_compatibility/test_cte_distributed.py::test_cte_distributed 9.66s call test_broken_projections/test.py::test_broken_ignored 9.45s call test_concurrent_queries_restriction_by_query_kind/test.py::test_insert 9.01s call test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] 8.18s call test_async_load_databases/test.py::test_multiple_tables 6.50s setup test_concurrent_queries_restriction_by_query_kind/test.py::test_insert 6.42s teardown test_backup_restore_on_cluster/test.py::test_tables_dependency 6.30s call test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] 6.00s setup test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column 5.94s teardown test_default_database_on_cluster/test.py::test_default_database_on_cluster 5.93s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 5.72s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] 5.32s teardown test_allowed_client_hosts/test.py::test_denied_host 5.28s call test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 4.84s teardown test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 4.78s teardown test_block_structure_mismatch/test.py::test 4.68s teardown test_covered_by_broken_exists/test.py::test_make_clone_covered_by_broken_detached_dir_exists 4.67s teardown test_authentication/test.py::test_authentication_pass 4.62s teardown test_buffer_profile/test.py::test_default_profile 4.57s teardown test_concurrent_queries_restriction_by_query_kind/test.py::test_select 4.51s call test_cleanup_after_start/test.py::test_old_dirs_cleanup 4.38s setup test_config_hide_in_preprocessed/test.py::test_hide_in_preprocessed 4.26s teardown test_backward_compatibility/test_cte_distributed.py::test_cte_distributed 4.08s call test_backup_restore_on_cluster/test.py::test_replicated_database_async 4.06s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] 4.05s teardown test_config_hide_in_preprocessed/test.py::test_hide_in_preprocessed 4.02s teardown test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 4.01s teardown test_broken_projections/test.py::test_materialize_broken_projection 3.96s call test_broken_projections/test.py::test_broken_projections_in_backups_2 3.94s call test_covered_by_broken_exists/test.py::test_make_clone_covered_by_broken_detached_dir_exists 3.89s teardown test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column 3.87s call test_backup_restore_on_cluster/test.py::test_mutation 3.84s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] 3.80s teardown test_cleanup_after_start/test.py::test_old_dirs_cleanup 3.75s teardown test_default_role/test.py::test_wrong_set_default_role 3.57s call test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] 3.42s teardown test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 3.29s call test_broken_projections/test.py::test_broken_projections_in_backups_1 3.19s call test_backup_restore_on_cluster/test.py::test_tables_dependency 3.18s call test_attach_table_normalizer/test.py::test_attach_substr_restart 3.14s teardown test_attach_table_normalizer/test.py::test_attach_substr_restart 2.86s call test_broken_projections/test.py::test_broken_projections_in_backups_3 2.76s call test_broken_projections/test.py::test_broken_ignored_replicated 2.69s teardown test_async_load_databases/test.py::test_multiple_tables 2.59s teardown test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 2.46s call test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 2.31s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 2.27s call test_backup_restore_on_cluster/test.py::test_required_privileges 2.15s call test_backup_restore_on_cluster/test.py::test_replicated_database 2.15s call test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 2.13s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] 2.07s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried 2.04s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] 2.03s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] 2.03s call test_broken_projections/test.py::test_materialize_broken_projection 2.03s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] 1.99s call test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries 1.98s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] 1.93s call test_backup_restore_on_cluster/test.py::test_projection 1.93s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] 1.88s call test_backup_restore_on_cluster/test.py::test_file_deduplication 1.83s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] 1.74s call test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 1.60s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 1.59s call test_attach_table_from_s3_plain_readonly/test.py::test_attach_table_from_s3_plain_readonly 1.55s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 1.54s call test_backup_restore_on_cluster/test.py::test_replicated_table 1.49s call test_default_role/test.py::test_alter_user 1.42s call test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 1.41s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 1.29s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 1.27s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 1.26s call test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 1.22s call test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 1.19s call test_backup_restore_on_cluster/test.py::test_system_users 1.18s call test_allowed_client_hosts/test.py::test_allowed_host 1.17s call test_default_database_on_cluster/test.py::test_default_database_on_cluster 1.16s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 1.10s call test_backup_restore_on_cluster/test.py::test_system_functions 1.01s call test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 0.97s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 0.95s call test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 0.89s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] 0.85s call test_default_role/test.py::test_set_default_roles 0.84s call test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 0.84s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] 0.82s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] 0.81s call test_backup_restore_on_cluster/test.py::test_empty_replicated_table 0.77s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 0.76s call test_attach_table_normalizer/test.py::test_attach_substr 0.73s call test_config_hide_in_preprocessed/test.py::test_hide_in_preprocessed 0.72s teardown test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 0.71s teardown test_backup_restore_on_cluster/test.py::test_replicated_table 0.71s teardown test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] 0.68s call test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column 0.67s teardown test_backup_restore_on_cluster/test.py::test_mutation 0.65s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] 0.64s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] 0.62s teardown test_backup_restore_on_cluster/test.py::test_replicated_database 0.62s teardown test_backup_restore_on_cluster/test.py::test_projection 0.62s teardown test_backup_restore_on_cluster/test.py::test_replicated_database_async 0.62s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 0.61s teardown test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 0.61s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 0.61s teardown test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 0.61s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 0.61s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] 0.60s call test_attach_backup_from_s3_plain/test.py::test_attach_part[wide0] 0.58s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] 0.57s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] 0.57s call test_authentication/test.py::test_authentication_pass 0.57s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 0.56s teardown test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 0.56s teardown test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 0.56s teardown test_backup_restore_on_cluster/test.py::test_system_functions 0.56s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 0.56s teardown test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 0.56s teardown test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 0.56s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 0.56s call test_allowed_client_hosts/test.py::test_denied_host 0.55s call test_attach_backup_from_s3_plain/test.py::test_attach_part[compact0] 0.52s teardown test_backup_restore_on_cluster/test.py::test_system_users 0.51s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 0.51s teardown test_backup_restore_on_cluster/test.py::test_empty_replicated_table 0.51s teardown test_backup_restore_on_cluster/test.py::test_file_deduplication 0.51s teardown test_backup_restore_on_cluster/test.py::test_required_privileges 0.46s teardown test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] 0.38s call test_backward_compatibility/test_cte_distributed.py::test_cte_distributed 0.33s call test_default_role/test.py::test_wrong_set_default_role 0.30s call test_attach_backup_from_s3_plain/test.py::test_attach_part[compact1] 0.30s call test_block_structure_mismatch/test.py::test 0.30s call test_attach_backup_from_s3_plain/test.py::test_attach_part[wide1] 0.25s call test_authentication/test.py::test_authentication_fail 0.25s call test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 0.18s call test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 0.18s call test_buffer_profile/test.py::test_buffer_profile 0.14s call test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 0.13s call test_buffer_profile/test.py::test_default_profile 0.11s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried 0.10s setup test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries 0.10s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] 0.08s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] 0.08s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] 0.08s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] 0.08s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] 0.08s setup test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 0.08s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] 0.07s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] 0.07s setup test_default_role/test.py::test_wrong_set_default_role 0.07s setup test_default_role/test.py::test_set_default_roles 0.00s teardown test_config_xml_main/test.py::test_xml_main_conf 0.00s teardown test_attach_backup_from_s3_plain/test.py::test_attach_part[compact0] 0.00s setup test_cgroup_limit/test.py::test_cgroup_cpu_limit 0.00s teardown test_concurrent_queries_restriction_by_query_kind/test.py::test_insert 0.00s setup test_attach_backup_from_s3_plain/test.py::test_attach_part[compact1] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] 0.00s teardown test_broken_projections/test.py::test_broken_ignored 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 0.00s setup test_attach_backup_from_s3_plain/test.py::test_attach_part[wide0] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database_async 0.00s teardown test_async_load_databases/test.py::test_dependent_tables 0.00s setup test_broken_projections/test.py::test_materialize_broken_projection 0.00s setup test_backup_restore_on_cluster/test.py::test_system_users 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] 0.00s setup test_async_load_databases/test.py::test_multiple_tables 0.00s setup test_broken_projections/test.py::test_broken_projections_in_backups_3 0.00s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] 0.00s setup test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 0.00s setup test_backup_restore_on_cluster/test.py::test_empty_replicated_table 0.00s setup test_authentication/test.py::test_authentication_pass 0.00s setup test_attach_backup_from_s3_plain/test.py::test_attach_part[wide1] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 0.00s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 0.00s setup test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] 0.00s setup test_backup_restore_on_cluster/test.py::test_projection 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table 0.00s setup test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 0.00s setup test_concurrent_queries_restriction_by_query_kind/test.py::test_select 0.00s setup test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] 0.00s setup test_backup_restore_on_cluster/test.py::test_required_privileges 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] 0.00s setup test_config_xml_main/test.py::test_xml_main_conf 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 0.00s setup test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] 0.00s teardown test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 0.00s setup test_backup_restore_on_cluster/test.py::test_mutation 0.00s setup test_backup_restore_on_cluster/test.py::test_system_functions 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 0.00s setup test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 0.00s setup test_backup_restore_on_cluster/test.py::test_file_deduplication 0.00s setup test_backup_restore_on_cluster/test.py::test_tables_dependency 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 0.00s setup test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 0.00s setup test_broken_projections/test.py::test_broken_projections_in_backups_1 0.00s setup test_broken_projections/test.py::test_broken_ignored_replicated 0.00s setup test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 0.00s teardown test_allowed_client_hosts/test.py::test_allowed_host 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] 0.00s setup test_async_load_databases/test.py::test_dict_get_data 0.00s teardown test_attach_backup_from_s3_plain/test.py::test_attach_part[compact1] 0.00s teardown test_async_load_databases/test.py::test_dict_get_data 0.00s setup test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 0.00s teardown test_broken_projections/test.py::test_broken_projections_in_backups_3 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] 0.00s teardown test_default_role/test.py::test_alter_user 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] 0.00s setup test_allowed_client_hosts/test.py::test_denied_host 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] 0.00s setup test_broken_projections/test.py::test_broken_projections_in_backups_2 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 0.00s teardown test_authentication/test.py::test_authentication_fail 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] 0.00s teardown test_default_role/test.py::test_set_default_roles 0.00s teardown test_cgroup_limit/test.py::test_cgroup_cpu_limit 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] 0.00s setup test_buffer_profile/test.py::test_default_profile 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] 0.00s teardown test_broken_projections/test.py::test_broken_projections_in_backups_2 0.00s teardown test_broken_projections/test.py::test_broken_ignored_replicated 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] 0.00s teardown test_attach_table_normalizer/test.py::test_attach_substr 0.00s teardown test_broken_projections/test.py::test_broken_projections_in_backups_1 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries 0.00s teardown test_buffer_profile/test.py::test_buffer_profile 0.00s setup test_attach_table_normalizer/test.py::test_attach_substr_restart 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] 0.00s teardown test_attach_backup_from_s3_plain/test.py::test_attach_part[wide0] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] =========================== short test summary info ============================ FAILED test_backup_restore_on_cluster/test.py::test_replicated_database - AssertionError: assert '' == 1 Don\'t\n2 count\n3 your\n4 chickens FAILED test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts PASSED test_attach_table_normalizer/test.py::test_attach_substr PASSED test_authentication/test.py::test_authentication_fail PASSED test_authentication/test.py::test_authentication_pass PASSED test_default_role/test.py::test_alter_user PASSED test_default_role/test.py::test_set_default_roles PASSED test_default_role/test.py::test_wrong_set_default_role PASSED test_attach_table_normalizer/test.py::test_attach_substr_restart PASSED test_attach_backup_from_s3_plain/test.py::test_attach_part[compact0] PASSED test_attach_backup_from_s3_plain/test.py::test_attach_part[compact1] PASSED test_attach_backup_from_s3_plain/test.py::test_attach_part[wide0] PASSED test_attach_backup_from_s3_plain/test.py::test_attach_part[wide1] PASSED test_allowed_client_hosts/test.py::test_allowed_host PASSED test_allowed_client_hosts/test.py::test_denied_host PASSED test_async_load_databases/test.py::test_dependent_tables PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] PASSED test_broken_projections/test.py::test_broken_ignored PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] PASSED test_broken_projections/test.py::test_broken_ignored_replicated PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] PASSED test_buffer_profile/test.py::test_buffer_profile PASSED test_buffer_profile/test.py::test_default_profile PASSED test_broken_projections/test.py::test_broken_projections_in_backups_1 PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] PASSED test_concurrent_queries_restriction_by_query_kind/test.py::test_insert PASSED test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] PASSED test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica PASSED test_broken_projections/test.py::test_broken_projections_in_backups_2 PASSED test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] PASSED test_backup_restore_on_cluster/test.py::test_empty_replicated_table PASSED test_broken_projections/test.py::test_broken_projections_in_backups_3 PASSED test_backup_restore_on_cluster/test.py::test_file_deduplication PASSED test_backup_restore_on_cluster/test.py::test_get_error_from_other_host PASSED test_broken_projections/test.py::test_materialize_broken_projection PASSED test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] PASSED test_backup_restore_on_cluster/test.py::test_keeper_value_max_size PASSED test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries PASSED test_attach_table_from_s3_plain_readonly/test.py::test_attach_table_from_s3_plain_readonly PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] PASSED test_backup_restore_on_cluster/test.py::test_mutation PASSED test_async_load_databases/test.py::test_dict_get_data PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] PASSED test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] PASSED test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible PASSED test_backward_compatibility/test_cte_distributed.py::test_cte_distributed PASSED test_backup_restore_on_cluster/test.py::test_projection PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] PASSED test_cgroup_limit/test.py::test_cgroup_cpu_limit PASSED test_concurrent_queries_restriction_by_query_kind/test.py::test_select PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] PASSED test_async_load_databases/test.py::test_multiple_tables PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_async PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] PASSED test_config_hide_in_preprocessed/test.py::test_hide_in_preprocessed PASSED test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column PASSED test_backup_restore_on_cluster/test.py::test_replicated_table PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] PASSED test_config_xml_main/test.py::test_xml_main_conf PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster PASSED test_default_database_on_cluster/test.py::test_default_database_on_cluster PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge PASSED test_detached_parts_metrics/test.py::test_numbers_of_detached_parts PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried PASSED test_backup_restore_on_cluster/test.py::test_required_privileges PASSED test_cleanup_after_start/test.py::test_old_dirs_cleanup PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] PASSED test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] PASSED test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts PASSED test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] PASSED test_backup_restore_on_cluster/test.py::test_system_functions PASSED test_backup_restore_on_cluster/test.py::test_system_users PASSED test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def PASSED test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] PASSED test_backup_restore_on_cluster/test.py::test_tables_dependency PASSED test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] PASSED test_covered_by_broken_exists/test.py::test_make_clone_covered_by_broken_detached_dir_exists PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] PASSED test_block_structure_mismatch/test.py::test =================== 2 failed, 98 passed in 156.68s (0:02:36) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 437, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_c5xcdm --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1d1c87eee2db -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_allowed_client_hosts/test.py::test_allowed_host test_allowed_client_hosts/test.py::test_denied_host test_async_load_databases/test.py::test_dependent_tables test_async_load_databases/test.py::test_dict_get_data test_async_load_databases/test.py::test_multiple_tables 'test_attach_backup_from_s3_plain/test.py::test_attach_part[compact0]' 'test_attach_backup_from_s3_plain/test.py::test_attach_part[compact1]' 'test_attach_backup_from_s3_plain/test.py::test_attach_part[wide0]' 'test_attach_backup_from_s3_plain/test.py::test_attach_part[wide1]' test_attach_table_from_s3_plain_readonly/test.py::test_attach_table_from_s3_plain_readonly test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart test_authentication/test.py::test_authentication_fail test_authentication/test.py::test_authentication_pass test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True]' test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes test_backup_restore_on_cluster/test.py::test_empty_replicated_table test_backup_restore_on_cluster/test.py::test_file_deduplication test_backup_restore_on_cluster/test.py::test_get_error_from_other_host test_backup_restore_on_cluster/test.py::test_keeper_value_max_size test_backup_restore_on_cluster/test.py::test_mutation test_backup_restore_on_cluster/test.py::test_projection test_backup_restore_on_cluster/test.py::test_replicated_database test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts test_backup_restore_on_cluster/test.py::test_replicated_table test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath test_backup_restore_on_cluster/test.py::test_required_privileges 'test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False]' 'test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True]' test_backup_restore_on_cluster/test.py::test_system_functions test_backup_restore_on_cluster/test.py::test_system_users test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty test_backup_restore_on_cluster/test.py::test_tables_dependency test_backward_compatibility/test_cte_distributed.py::test_cte_distributed test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts test_block_structure_mismatch/test.py::test test_broken_projections/test.py::test_broken_ignored test_broken_projections/test.py::test_broken_ignored_replicated test_broken_projections/test.py::test_broken_projections_in_backups_1 test_broken_projections/test.py::test_broken_projections_in_backups_2 test_broken_projections/test.py::test_broken_projections_in_backups_3 test_broken_projections/test.py::test_materialize_broken_projection test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile test_cgroup_limit/test.py::test_cgroup_cpu_limit 'test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node]' 'test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries]' test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd]' test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True]' test_cleanup_after_start/test.py::test_old_dirs_cleanup test_concurrent_queries_restriction_by_query_kind/test.py::test_insert test_concurrent_queries_restriction_by_query_kind/test.py::test_select test_config_hide_in_preprocessed/test.py::test_hide_in_preprocessed test_config_xml_main/test.py::test_xml_main_conf test_covered_by_broken_exists/test.py::test_make_clone_covered_by_broken_detached_dir_exists test_default_database_on_cluster/test.py::test_default_database_on_cluster test_default_role/test.py::test_alter_user test_default_role/test.py::test_set_default_roles test_default_role/test.py::test_wrong_set_default_role test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed]' -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 ' returned non-zero exit status 1.